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Introduction 


This brief presents findings from a study on 
Urban Advantage (UA), a collaboration between 
the American Museum of Natural History 
(AMNH), other New York City informal science 
institutions, and the New York City Department 
of Education (NYCDOE) to improve the science 
literacy of NYC public school students. The 
following analysis draws on a rich longitudinal 
database, containing student- and school-level 
data for all NYC public schools and students 
from 2004-05 to 2009-10. Results indicate that 
these collaborations can have a positive impact 
on student achievement and science learning. 

The findings from this study are especially timely 
given that U.S. students consistently show low 
levels of achievement on tests of science literacy. 
On the 2009 National Assessment of Education 
Progress (NAEP), only 63% of eighth-graders 
demonstrated “partial mastery of the knowledge 


Key Findings 

• Students at UA schools outperform 
students at non-UA schools. In 2006- 
07, the third year of the program, 

44.2% of students at UA schools are 
proficient on the Intermediate Level 
Science Test (ILS) exam, compared to 
40.5% at non-UA schools. In 2008-09, 
55.5% of students at UA schools are 
proficient, compared to 46.2% of 
students at non-UA schools. 

• The magnitude of the difference 
between students at UA and non-UA 
schools increases over time. Little 
change is seen in student performance 
on ELA or math for eighth-grade 
students, suggesting the effect is not 
merely reflecting coincident overall 
school improvement. 

• UA has grown from 35 schools and 63 
teachers in 2005 to 174 schools and 
386 teachers in 2010 and serves 
approximately 35% of middle schools 
and 20% of students in NYC. 

• UA schools are similar to other NYC 
schools serving eighth graders, despite 
larger enrollments. 
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and skills that are fundamental for proficient work” in science (National Center for Education 
Statistics, 2011). For urban school districts like New York City (NYC), the NAEP scores are 
even more alarming: only 38% of NYC eighth-graders scored at or above this level of basic 
proficiency, with even lower scores for black and Hispanic students (24% and 26%, 
respectively) (National Center for Education Statistics, 2011). 

Given these low levels of science knowledge, programs such as UA are important for re- 
envisioning the approaches, leadership, and methodologies for educating American children in 
science. Within the formal education sector, researchers have pointed to the importance of 
supporting students’ college and career readiness by more closely aligning K-12 education 
standards with the knowledge and skills they will need to succeed in introductory college-level 
science courses (The College Board, 2009). At the same time, informal science education 
institutions seeking to support the science efforts of school systems that have increasingly 
focused their attention on reading/language arts and math have been identified as critical 
participants, “premised on the notion that their emphasis on phenomena-rich, learning-driven 
interactions with science resonates with the notion of inquiry underlying K-12 science education 
reform” (National Research Council, 2009). 

What is Urban Advantage? 


The primary goal of Urban Advantage (UA) is to improve student understanding of scientific 
inquiry as defined in the New York State Core Curriculum. UA was initiated in 2004 and is 
administered by the Gottesman Center for Science Teaching and Learning at the AMNH 
working in collaboration with several informal science institutions, including the Brooklyn Botanic 
Garden, New York Botanical Garden, New York Hall of Science, Queens Botanical Garden, 
Staten Island Zoological Society, the Wildlife Conservation Society’s Bronx Zoo and New York 
Aquarium, and the NYCDOE. The name “Urban Advantage” reflects the partners’ belief that it is 
an advantage to live in an urban setting with so many science-rich cultural institutions and 
nature facilities. UA differs fundamentally from traditional museum-to-school collaborations as it 
provides a hybrid model for civic engagement where the resources of institutions are selected, 
designed, and shaped by the specific alignment to the science curriculum of NYC’s middle 
schools. 
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UA provides teachers with 48 hours of professional development over eight days during their 
first year of the program and an additional 10 hours beginning in year two. Professional 
development emphasizes authentic hands-on learning experiences in science, the nature of 
scientific work, specific science topics, and the essential features of inquiry (NRC, 2000) in the 
form of long-term science investigations, referred to as science exit projects in NYC. After 
choosing a UA partner institution to work with, teachers learn how to plan effective field trips, 
embed resources in instruction, use UA-provided equipment and resources, teach students the 
components of experimental design, and to develop scientific explanations based on claims, 
evidence, and reasoning to help students complete the science exit projects. UA also provides 
free admission to all partner institutions in the form of vouchers to facilitate individual, class, and 
family field trips. Participating schools also receive transportation funds for weekend family 
trips, which are organized by each 
school’s parent coordinator 1 . 

As shown in Table 1, UA has 
substantially expanded since its 
first year of operation. It now 
serves schools in every community 
school and NYC council district. In 
its first year of operation, UA 
worked with 63 teachers in 35 
schools and reached 5500 
students. By year 4 UA was working with 257 teachers and 27,541 students in 155 schools; in 
the current school year (2010-11), the program serves 156 schools, 371 teachers, and 37,822 
students in grades 6-8, representing 35% of all middle schools with eighth-grade and 20% of 
students in grades 6-8. 

UA schools are, in many respects, quite similar to other New York City schools serving eighth- 
graders. The one consistent difference between UA and non-UA schools is size. On average, 
UA schools are larger than non-UA schools. And across UA schools, as with city schools as a 
whole, there is much variation. UA serves students in schools that vary in size, demographics, 
and poverty status. 


Table 1 : Growth of Urban Advantage from 2005 to 2010 



2005 

2006 

2007 

2008 

2009 

2010 


Y1 

Y2 

Y3 

Y4 

Y5 

Y6 

# UA 
Students 

5,500 

18,722 

21,016 

27,541 

24,793 

37,582 

# UA 
Schools 

31 

111 

129 

156 

147 

174 

# UA 
Teachers 

62 

195 

210 

256 

257 

386 


1 Parent coordinators are members of schools’ administrative teams and work with school staff, parent associations, 
and community groups to increase parent involvement in schools. 
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Results 


Figure 1 shows the distribution of 
eighth-grade students scoring in 
levels 3 or 4 on the New York State 
(NYS) Intermediate-Level Science 
(ILS) Test from years 2003-04 (one 
year prior to UA) through 2009-10. 

Prior to UA, less than 40% of NYC 
eighth-graders were proficient in 
science in 2003-04, considerably 
less than the NYS average of 86%. 

In the first two years of UA, no 
significant differences are found in performance between students at UA and non-UA schools. 
However, in the third year differences begin to emerge. In 2006-07, the difference between the 
two groups grows to 3.7% as students at UA schools begin to outperform students at non-UA 
schools on the ILS exam, 44.2% to 40.5%. The difference between the two groups expands in 
years four and five to over 9%, when over 55% of UA students are proficient, compared to less 
than 50% of students at non-UA schools. 

This finding is consistent with the literature of school improvement who believe that three years 
is the minimum amount of time needed to see results (Fullan & Stiegelbauer). By the third year 
of implementation (2007-08), UA had a more-developed and stronger program that included not 
only professional development but also a set of materials and resources to help teachers in the 
classroom. 

It is possible that selection may play a significant role in the success of the program. That is, is 
there a difference in the schools that choose to participate in UA? Schools and teachers that 
choose to enter or remain in UA may have higher performing students to begin with or students 
who differ in other ways that may influence performance than the schools that choose not to 
participate in UA. 


Figure 1. Percent of Students in Each Year who Meet Standards on 
eighth Grade ILS by UA and Non-UA Schools 
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While the number of schools that participate in UA differs in each year, Table 2 shows that 
schools that enter UA are no different than those which do not in the year prior to joining. None 
of the demographic characteristics of students or the percent of students who are proficient in 
ELA, mathematics or science show statistically significant differences between UA schools and 
non-UA schools. The only statistically significant difference is for total enrollment in the first 
year of UA when the schools were significantly larger than those not entering UA. 


Table 2: Mean Characteristics of UA and Non-UA Schools, Year Prior to Joining UA 


2004 2005 2006 2007 2008 2009 


N of Schools 

UA 

31 

Non- 

UA 

289 

UA 

61 

Non- 

UA 

291 

UA 

43 

Non- 

UA 

366 

UA 

42 

Non- 

UA 

244 

UA 

7 

Non- 

UA 

227 

UA 

24 

Non- 

UA 

238 

Total Enrollment 

1079 

851 

851 

784 

611 

586 

580 

539 

738 

667 

519 

621 


(434) 

(468) 

(500) 

(439) 

(426) 

(425) 

(439) 

(390) 

(647) 

(373) 

(328) 

(375) 

% Black 

41.84 

36.47 

34.05 

38.67 

37.65 

38.95 

36.95 

39.41 

33.89 

38.52 

35.74 

38.17 


(28.1) 

(28.1) 

(26.2) 

(29.5) 

(29.3) 

(28.9) 

(29.4) 

(29.3) 

(34.0) 

(29.3) 

(26.7) 

(29.4) 

% Hispanic 

35.10 

39.91 

42.81 

39.56 

42.21 

40.45 

43.27 

40.48 

35.56 

41.18 

46.87 

40.74 


(22.9) 

(25.4) 

(26.3) 

(25.7) 

(27.5) 

(26.0) 

(25.4) 

(26.4) 

(23.2) 

(26.4) 

(23.0) 

(25.9) 

% Asian/Other 

13.26 

9.67 

10.20 

9.42 

7.16 

8.77 

10.32 

8.61 

12.56 

9.14 

7.13 

10.37 


(19.6) 

(12.1) 

(12.9) 

(13.0) 

(12.0) 

(13.3) 

(14.2) 

(13.4) 

(14.5) 

(14.1) 

(9.7) 

(14.9) 

% White 

9.82 

13.96 

12.94 

12.34 

12.53 

11.18 

8.87 

10.95 

17.78 

10.71 

10.48 

10.64 


(18.2) 

(19.6) 

(19.3) 

(19.1) 

(22.1) 

(18.0) 

(13.3) 

(18.5) 

(19.1) 

(18.0) 

(14.4) 

(17.7) 

% ELL 

10.26 

10.60 

11.68 

10.59 

10.79 

10.80 

11.29 

10.74 

9.67 

11.23 

10.56 

12.02 


(7.8) 

(10.6) 

(10.7) 

(10.4) 

(9.9) 

(11.1) 

(10.2) 

(12.7) 

(4.2) 

(11.7) 

(10.1) 

(11.7) 

% Free Lunch 

75.37 

71.10 

69.20 

68.66 

63.30 

69.90 

64.76 

66.29 

55.89 

66.30 

72.48 

69.98 


(21.8) 

(23.5) 

(21.7) 

(22.5) 

(23.1) 

(23.3) 

(30.2) 

(27.1) 

(31.5) 

(25.8) 

(15.8) 

(20.4) 

% Prof. ELA 

33.17 

39.42 

50.94 

46.94 

36.11 

40.14 

42.32 

42.37 

58.41 

48.75 

43.17 

41.71 


(16.6) 

(20.5) 

(19.9) 

(21.3) 

(20.4) 

(21.3) 

(19.1) 

(21.6) 

(18.9) 

(21.6) 

(22.1) 

(20.2) 

% Prof. Math 

38.10 

43.63 

48.49 

45.07 

36.34 

43.08 

53.53 

50.71 

73.27 

62.42 

48.64 

48.77 


(17.4) 

(20.6) 

(21.7) 

(22.3) 

(23.2) 

(22.2) 

(21.2) 

(23.6) 

(18.7) 

(23.5) 

(25.6) 

(22.3) 

% Prof. Science 

38.23 

45.03 

46.88 

45.17 

36.61 

39.52 

43.00 

42.05 

47.57 

50.16 

45.34 

47.86 


(20.9) 

(24.8) 

(23.9) 

(23.8) 

(23.0) 

(24.0) 

(19.6) 

(24.6) 

(27.7) 

(22.7) 

(28.4) 

(23.0) 


Standard deviations are in parentheses 

Bold indicates differences are statistically significant at .05 level or less 
% Proficient is the percent scoring in levels 3 or 4 


As shown in Table 3, controlling for the year prior to becoming UA, we find that UA students, on 
average, do 0.041 standard deviations better than students at non-UA schools (Model 1). If we 
control for the first year that schools are in UA, students at UA schools do even better in the 
years post UA entry, with those students performing at .056 standard deviations higher than 
students at non-UA schools (Model 2). While black and Hispanic students do worse compared 
to white students in science, we do find that UA has some impact in reducing the disparity for 
black students (Model 3). We find that black students at UA schools, in general, score 0.066 
standard deviations better than black students at non-UA students. Asian students, who 
generally outperform white students, do perform even better if they are in UA schools. On the 
other hand, female students perform worse when compared to male students, and it appears 
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that, in general, female students at UA schools do significantly worse than their non-UA 
counterparts (P=-0.033). 

The same analyses were completed using ELA and math as the dependent variables; this is a 
check to see if, in general, students at UA schools are higher performing than students at non- 
UA schools. No significant differences are found for math or ELA. This provides evidence that 
UA has significant impacts for students on the eighth-grade science exam. 


Table 3: OLS Regression, Urban Advantage, 2005 through 2010 



Science 

Math 

ELA 

Science 


Model 1 

Model 2 

Model 2 

Model 2 

Model 3 


P/se 

P/se 

P/se 

P/se 

b/se 

Year Prior UA 

0.002 

0.011 

0.011 

-0.001 

0.010 


(0.018) 

(0.021) 

(0.024) 

(0.017) 

(0.021) 

UA in Any Year 

0.041* 






(0.016) 





Year Ent. UA 


0.044 

0.036 

0.026 

0.013 



(0.024) 

(0.027) 

(0.021) 

(0.038) 

Years Post UA 


0.056* 

0.014 

0.022 

0.030 



(0.028) 

(0.031) 

(0.023) 

(0.037) 

Black 

-0.397*** 

-0.397*** 

-0.408*** 

-0.375*** 

-0.411*** 


(0.017) 

(0.017) 

(0.022) 

(0.023) 

(0.017) 

Hispanic 

-0.226*** 

-0.226*** 

-0.270*** 

-0.275*** 

-0.235*** 


(0.015) 

(0.015) 

(0.020) 

(0.021) 

(0.015) 

Asian 

0.162*** 

0.162*** 

0.407*** 

0.064** 

0.145*** 


(0.019) 

(0.019) 

(0.027) 

(0.022) 

(0.021) 

Female 

-0.072*** 

-0.072*** 

0.027*** 

0.194*** 

-0.062*** 


(0.005) 

(0.005) 

(0.004) 

(0.004) 

(0.005) 

UA*White 





0.012 






(0.028) 

UA*Black 





0.062* 






(0.030) 

UA*Hispanic 





0.042 






(0.027) 

U A* Asian 





0.066* 






(0.032) 

UA*Female 





-0.033*** 






(0.008) 

Constant 

41.407*** 

45.897*** 

21.546* 

4.157 

44.951*** 


(6.584) 

(8.097) 

(8.614) 

(7.134) 

(8.153) 

School FE 

YES 

YES 

YES 

YES 

YES 

R-Square 

0.35 

0.35 

0.33 

0.32 

0.35 

N 

401270 

401270 

425820 

409572 

401270 


(1) * p<0.05, ** p<0.01, *** p<0.001 

(2) Robust clustered standard errors in parentheses 

(3) LEP, Special Education, Poverty, and Year dummies not shown 
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Summary 


Our study provides the first estimates of the impact of Urban Advantage on eighth-grade test 
scores for NYC students. In short, we find evidence that UA improves performance in science: 
student performance on the NYS science exam increases with the implementation of UA and 
the magnitude of the difference between UA and non-UA schools increases over time. Little 
change in seen in student performance on ELA or math for eighth-grade students, suggesting 
the effect is not merely reflecting coincident overall school improvement. Exploratory subgroup 
analyses suggest the impact is largest for black students, and less successful for girls than 
boys. 

Recent publications (NRC, 2005; 2007) indicate that, if students are to understand science, they 
must have opportunities to do science. At its core, scientific inquiry involves conducting 
investigations by posing scientifically oriented questions, prioritizing evidence, and developing 
logical explanations. Local, state, national, and international science standards all recommend 
inquiry as a method to approach science instruction (American Association for the Advancement 
of Science, 1993; National Research Council, 1996; New York State Education Department, 

201 0; New York City Department of Education, 201 1 ). However, much of the current science 
instruction in schools in the United States does not utilize inquiry, instead taking a more 
simplistic approach (National Research Council, 2009). 

While schools traditionally have not utilized inquiry methods, many informal science institutions 
embrace this form of instruction. Informal science institutions are also believed to help make 
science learning “personally relevant and rewarding” (National Research Council, 2009, p. 1). 
Although schools and informal science institutions approach science differently and play 
different roles in students’ lives in regards to science instruction, there is a growing body of 
literature that points to the benefits of collaboration between formal and informal science 
institutions. For example, a National Research Council report (2009) on learning science in 
informal environments showed that informal science learning experiences impact children’s 
interest in science; in the report, the NRC recommended a collaboration between formal and 
informal education institutions to increase students’ science learning. Research shows that 
informal science institutions can be effective resources for hands-on science learning for 
students and increased science pedagogical content knowledge for teachers (Aquino, Kelly, & 
Bayne, 2010). Other studies have shown that in many circumstances, field trips can be a 
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beneficial addition to science instruction, though they have often been underused as teaching 
tools (DeWitt & Storksdieck, 2008). 

The results of this analysis give support to the call by the National Research Council and others 
for an increased role of informal institutions in science learning both for students and teachers. 
However, these institutions cannot work on their own. Strong partnerships between the 
institutions within each community and between the institution and the school district(s) in which 
they work (and in most cases, consistent funding sources) must be in place to enable these 
programs to grow and flourish and to provide students with the resources they need to develop 
as scientists. 
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